Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support #4695

Merged
merged 2 commits into from Feb 12, 2019

Conversation

ryanyuan
Copy link
Contributor

@ryanyuan ryanyuan commented Feb 12, 2019

BigQueryHook.run_with_configuration creates a new job and receives a response with location field. It should use that field for the keep_polling_job loop. Otherwise, it returns 404 job not found if the location is not US or EU.

After inserting new job and get the resp of the job details, it should retrieve the location and use it for the jobs.get request.

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow-3874 issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:
    test_bigquery_hook.py::TestBigQueryHookRunWithConfiguration

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

  • Passes flake8

…support

Assign location from the resp of insert() back to job.get()
@ryanyuan ryanyuan closed this Feb 12, 2019
@ryanyuan ryanyuan reopened this Feb 12, 2019
@codecov-io
Copy link

Codecov Report

Merging #4695 into master will increase coverage by 0.04%.
The diff coverage is 75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4695      +/-   ##
==========================================
+ Coverage   74.35%   74.39%   +0.04%     
==========================================
  Files         430      430              
  Lines       27964    27967       +3     
==========================================
+ Hits        20792    20807      +15     
+ Misses       7172     7160      -12
Impacted Files Coverage Δ
airflow/contrib/hooks/bigquery_hook.py 60.26% <75%> (+2.19%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a738e5...1dea85d. Read the comment docs.

Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Fokko Fokko merged commit 1c06c74 into apache:master Feb 12, 2019
@ryanyuan
Copy link
Contributor Author

ryanyuan commented Feb 12, 2019

LGTM according to the docs: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs

Thanks for the review @Fokko

@Fokko
Copy link
Contributor

Fokko commented Feb 27, 2019

My pleasure @ryanyuan

ashb pushed a commit that referenced this pull request Apr 26, 2019
… support (#4695)

* [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support

Assign location from the resp of insert() back to job.get()

* [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support

Remove unused code

(cherry picked from commit 1c06c74)
ashb pushed a commit to ashb/airflow that referenced this pull request Apr 29, 2019
… support (apache#4695)

* [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support

Assign location from the resp of insert() back to job.get()

* [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support

Remove unused code

(cherry picked from commit 1c06c74)
wmorris75 pushed a commit to modmed/incubator-airflow that referenced this pull request Jul 29, 2019
… support (apache#4695)

* [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support

Assign location from the resp of insert() back to job.get()

* [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support

Remove unused code
dharamsk pushed a commit to postmates/airflow that referenced this pull request Aug 8, 2019
… support (apache#4695)

* [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support

Assign location from the resp of insert() back to job.get()

* [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support

Remove unused code

(cherry picked from commit 1c06c74)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants